---
title: "Notes about BeautifulSoup"
date: 2019-3-20
categories:
- python
tags:
---

<div id="content">
<div id="table-of-contents">
<h2>Table of Contents</h2>
<div id="text-table-of-contents">
<ul>
<li><a href="#org45c72f9">查</a>
<ul>
<li><a href="#org52ab576">过虑器</a>
<ul>
<li><a href="#orga45a985">字符串</a></li>
<li><a href="#orge95025c">正则</a></li>
<li><a href="#org3a47768">列表</a></li>
<li><a href="#org760084d">True</a></li>
<li><a href="#org9b14a4a">Function 函数过虑器</a></li>
</ul>
</li>
<li><a href="#orge3a2486">find_all 最常用的查找方法</a>
<ul>
<li><a href="#org54d20d0">find_all 的简写</a></li>
</ul>
</li>
<li><a href="#orge7ae38f">decode_contents 相当于 innerHTML</a></li>
</ul>
</li>
<li><a href="#org7f847ec">树的修改</a>
<ul>
<li><a href="#org3b5ed85">extract 移除整个tag</a></li>
</ul>
</li>
</ul>
</div>
</div>
<div class="outline-2" id="outline-container-org45c72f9">
<h2 id="org45c72f9">查</h2>
<div class="outline-text-2" id="text-org45c72f9">
</div>
<div class="outline-3" id="outline-container-org52ab576">
<h3 id="org52ab576">过虑器</h3>
<div class="outline-text-3" id="text-org52ab576">
<p>
当搜索时，通常会指定 key=value 这样的条件，过虑是value部分，bs支持的查询过虑器有：字符串，正则，列表
</p>
</div>
<div class="outline-4" id="outline-container-orga45a985">
<h4 id="orga45a985">字符串</h4>
<div class="outline-text-4" id="text-orga45a985">
<p>
最简单的过虑器，例如 {% raw %} soup.find_all('b') {% endraw %} (key 是find_all方法的name参数，过虑器是'b'这个字符串）
</p>
</div>
</div>
<div class="outline-4" id="outline-container-orge95025c">
<h4 id="orge95025c">正则</h4>
<div class="outline-text-4" id="text-orge95025c">
<p>
例如：
</p>
<div class="org-src-container">
<pre class="src src-python"><span style="font-weight: bold;">import</span> re
<span style="font-weight: bold;">for</span> tag <span style="font-weight: bold;">in</span> soup.find_all(re.<span style="font-weight: bold;">compile</span>(<span style="font-style: italic;">"^b"</span>)):
    <span style="font-weight: bold;">print</span>(tag.name)
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">body</span>
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">b</span>
</pre>
</div>
</div>
</div>
<div class="outline-4" id="outline-container-org3a47768">
<h4 id="org3a47768">列表</h4>
<div class="outline-text-4" id="text-org3a47768">
<div class="org-src-container">
<pre class="src src-python">soup.find_all([<span style="font-style: italic;">"a"</span>, <span style="font-style: italic;">"b"</span>])
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">[&lt;b&gt;The Dormouse's story&lt;/b&gt;,</span>
<span style="font-weight: bold; font-style: italic;">#  </span><span style="font-weight: bold; font-style: italic;">&lt;a class="sister" href="http://example.com/elsie" id="link1"&gt;Elsie&lt;/a&gt;,</span>
<span style="font-weight: bold; font-style: italic;">#  </span><span style="font-weight: bold; font-style: italic;">&lt;a class="sister" href="http://example.com/lacie" id="link2"&gt;Lacie&lt;/a&gt;,</span>
<span style="font-weight: bold; font-style: italic;">#  </span><span style="font-weight: bold; font-style: italic;">&lt;a class="sister" href="http://example.com/tillie" id="link3"&gt;Tillie&lt;/a&gt;]</span>
</pre>
</div>
</div>
</div>
<div class="outline-4" id="outline-container-org760084d">
<h4 id="org760084d">True</h4>
<div class="outline-text-4" id="text-org760084d">
<p>
下面的示例返回所有的tags，除了文本。
匹配的是 name=True，name是tag的name，而每个tag肯定是有个name的，在进行测试的时候会返回True，所以所有的tag被返回了。
下面有这种过虑器的更多使用，例如 {% raw %} find_all(id = True) {% endraw %} 查询所有有id的tag
</p>
<div class="org-src-container">
<pre class="src src-python"><span style="font-weight: bold;">for</span> tag <span style="font-weight: bold;">in</span> soup.find_all(<span style="font-weight: bold; text-decoration: underline;">True</span>):
    <span style="font-weight: bold;">print</span>(tag.name)
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">html</span>
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">head</span>
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">title</span>
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">body</span>
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">p</span>
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">b</span>
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">p</span>
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">a</span>
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">a</span>
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">a</span>
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">p</span>
</pre>
</div>
</div>
</div>
<div class="outline-4" id="outline-container-org9b14a4a">
<h4 id="org9b14a4a">Function 函数过虑器</h4>
<div class="outline-text-4" id="text-org9b14a4a">
<p>
定义一个函数接收一个element元素，返回True or False。例如：
</p>
<div class="org-src-container">
<pre class="src src-python"><span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">函数被调用时，收到的tag参数是tag</span>
<span style="font-weight: bold;">def</span> <span style="font-weight: bold;">has_class_but_no_id</span>(tag):
    <span style="font-weight: bold;">return</span> tag.has_attr(<span style="font-style: italic;">'class'</span>) <span style="font-weight: bold;">and</span> <span style="font-weight: bold;">not</span> tag.has_attr(<span style="font-style: italic;">'id'</span>)

soup.find_all(has_class_but_no_id)
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">[&lt;p class="title"&gt;&lt;b&gt;The Dormouse's story&lt;/b&gt;&lt;/p&gt;,</span>
<span style="font-weight: bold; font-style: italic;">#  </span><span style="font-weight: bold; font-style: italic;">&lt;p class="story"&gt;Once upon a time there were...&lt;/p&gt;,</span>
<span style="font-weight: bold; font-style: italic;">#  </span><span style="font-weight: bold; font-style: italic;">&lt;p class="story"&gt;...&lt;/p&gt;]</span>
</pre>
</div>
<p>
如果过虑的不是tag，面是某个属性：例如 {% raw %} soup.find_all(href=not_lacie) {% endraw %} ,函数中收到的对数将是href的值，而不是整个tag
</p>
</div>
</div>
</div>
<div class="outline-3" id="outline-container-orge3a2486">
<h3 id="orge3a2486">find_all 最常用的查找方法</h3>
<div class="outline-text-3" id="text-orge3a2486">
<p>
{% raw %} find_all(name, attrs, recursive, string, limit, **kwargs) {% endraw %}
</p>
<dl class="org-dl">
<dt>name</dt><dd>如果指定了这个参数，则相当于document.getElementsByTagName，例如：</dd>
</dl>
<div class="org-src-container">
<pre class="src src-python">soup.find_all(<span style="font-style: italic;">"title"</span>) <span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">document.getElementsByTagName('title')</span>
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">[&lt;title&gt;The Dormouse's story&lt;/title&gt;]</span>
</pre>
</div>
<p>
name参数的值能够是上面所说的所有过虑器
</p>
<dl class="org-dl">
<dt>kwargs</dt><dd>这个方法不识别的参数会被当作keyword收集起来，用于过虑tag的attribute。例如：</dd>
</dl>
<div class="org-src-container">
<pre class="src src-python">soup.find_all(<span style="font-weight: bold;">id</span>=<span style="font-style: italic;">'link2'</span>) <span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">查找所有id=link2的tag</span>
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">[&lt;a class="sister" href="http://example.com/lacie" id="link2"&gt;Lacie&lt;/a&gt;]</span>
soup.find_all(href=re.<span style="font-weight: bold;">compile</span>(<span style="font-style: italic;">"elsie"</span>)) <span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">查找所有href匹配正则的标签</span>
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">[&lt;a class="sister" href="http://example.com/elsie" id="link1"&gt;Elsie&lt;/a&gt;]</span>
soup.find_all(<span style="font-weight: bold;">id</span>=<span style="font-weight: bold; text-decoration: underline;">True</span>) <span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">查找所有有id属性的tag</span>
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">[&lt;a class="sister" href="http://example.com/elsie" id="link1"&gt;Elsie&lt;/a&gt;,</span>
<span style="font-weight: bold; font-style: italic;">#  </span><span style="font-weight: bold; font-style: italic;">&lt;a class="sister" href="http://example.com/lacie" id="link2"&gt;Lacie&lt;/a&gt;,</span>
<span style="font-weight: bold; font-style: italic;">#  </span><span style="font-weight: bold; font-style: italic;">&lt;a class="sister" href="http://example.com/tillie" id="link3"&gt;Tillie&lt;/a&gt;]</span>
</pre>
</div>
<p>
支持上述所有过虑器。
Note：像 &lt;input name="email" /&gt; 这样的tag，如果用 {% raw %} find_all(name = "email") {% endraw %} 是找不到这个input tag的，因为name是find_all方法的参数名。需要用到下面的attrs方法参数。
</p>
<ul class="org-ul">
<li>通过css class查找</li>
</ul>
<p>
class是python关键字，所以要使用 {% raw %} class_ {% endraw %} 来过虑tag的css class。
</p>
<p>
一个tag也许有多个class，bs的行为很智能，是将它们作为多个class，而不是一整个字符串。
例如&lt;p class="body strikeout"&gt;&lt;/p&gt;，通过class_=body或者class_=strikeout都能找到这个tag，而像 {% raw %} class_="strikeout body") {% endraw %} 这样是不行的，应当使用列表作为过虑器。
</p>
<p>
也能使用css选择器的方式：
</p>
<div class="org-src-container">
<pre class="src src-python">css_soup.select(<span style="font-style: italic;">"p.strikeout.body"</span>)
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">[&lt;p class="body strikeout"&gt;&lt;/p&gt;]</span>
</pre>
</div>
<dl class="org-dl">
<dt>attrs</dt><dd>有的attribute，例如html5的data-*，中间有横线，如果 {% raw %} find_all(data-foo = "value") {% endraw %} 这么写，python是无法解释的，所以需要用到attrs参数</dd>
</dl>
<div class="org-src-container">
<pre class="src src-python">data_soup.find_all(attrs={<span style="font-style: italic;">"data-foo"</span>: <span style="font-style: italic;">"value"</span>})
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">[&lt;div data-foo="value"&gt;foo!&lt;/div&gt;]</span>
</pre>
</div>
<dl class="org-dl">
<dt>string</dt><dd>用于搜索 innerText 。在4.4.0之前是参数名是text。例如：</dd>
</dl>
<div class="org-src-container">
<pre class="src src-python">soup.find_all(string=<span style="font-style: italic;">"Elsie"</span>)
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">[u'Elsie']</span>

soup.find_all(string=[<span style="font-style: italic;">"Tillie"</span>, <span style="font-style: italic;">"Elsie"</span>, <span style="font-style: italic;">"Lacie"</span>])
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">[u'Elsie', u'Lacie', u'Tillie']</span>

soup.find_all(string=re.<span style="font-weight: bold;">compile</span>(<span style="font-style: italic;">"Dormouse"</span>))
[u<span style="font-style: italic;">"The Dormouse's story"</span>, u<span style="font-style: italic;">"The Dormouse's story"</span>]

<span style="font-weight: bold;">def</span> <span style="font-weight: bold;">is_the_only_string_within_a_tag</span>(s):
    <span style="font-style: italic;">"""Return True if this string is the only child of its parent tag."""</span>
    <span style="font-weight: bold;">return</span> (s == s.parent.string)

soup.find_all(string=is_the_only_string_within_a_tag)
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">[u"The Dormouse's story", u"The Dormouse's story", u'Elsie', u'Lacie', u'Tillie', u'...']</span>

soup.find_all(<span style="font-style: italic;">"a"</span>, string=<span style="font-style: italic;">"Elsie"</span>)
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">[&lt;a href="http://example.com/elsie" class="sister" id="link1"&gt;Elsie&lt;/a&gt;]</span>
</pre>
</div>
<dl class="org-dl">
<dt>limit</dt><dd>相当于sql中的limit，当文档很大的时候，find_all会耗费时间，如果不需要全部的结果则指定limit参数</dd>
<dt>recursive</dt><dd>默认bs会查询所有的了结点，如果设置为False，则只查询直系子结点。</dd>
</dl>
</div>
<div class="outline-4" id="outline-container-org54d20d0">
<h4 id="org54d20d0">find_all 的简写</h4>
<div class="outline-text-4" id="text-org54d20d0">
<p>
因为这个方法用得特别频繁，所以：
</p>
<div class="org-src-container">
<pre class="src src-python">soup.find_all(<span style="font-style: italic;">"a"</span>)
soup(<span style="font-style: italic;">"a"</span>)
</pre>
</div>
<p>
这两种写法是等价的。
</p>
</div>
</div>
</div>
<div class="outline-3" id="outline-container-orge7ae38f">
<h3 id="orge7ae38f">decode_contents 相当于 innerHTML</h3>
<div class="outline-text-3" id="text-orge7ae38f">
<p>
Since 4.0.1，这个函数并没有出现在官网的文档中。
Renders the contents of this tag as a Unicode string.
将当前标签渲染成unicode字符串。
</p>
<dl class="org-dl">
<dt>{% raw %} indent_level {% endraw %}</dt><dd>Each line of the rendering will be indented this many spaces 缩进的空格数</dd>
<dt>{% raw %} eventual_encoding {% endraw %}</dt><dd>The tag is destined to be encoded into this encoding. 指的是取代&lt;META&gt;中的编码，而不是渲染时使用的编码</dd>
</dl>
<p>
This method is <span class="underline">not</span> responsible for performing that encoding. 
This information is passed in so that it can be substituted in if the contains a &lt;META&gt; tag that mentions the document's encoding.
</p>
<dl class="org-dl">
<dt>{% raw %} formatter {% endraw %}</dt><dd>The output formatter responsible for converting entities to Unicode characters. formatter参考官网文档</dd>
</dl>
<p>
另外还有函数 {% raw %} encode_contents {% endraw %} (since 4.0.4)，将当前标签渲染成bytestring。
</p>
<p>
bs3中使用 {% raw %} renderContents {% endraw %} 函数。这个函数在4.0.4之后又被重新引入，以保持兼容性。
</p>
</div>
</div>
</div>
<div class="outline-2" id="outline-container-org7f847ec">
<h2 id="org7f847ec">树的修改</h2>
<div class="outline-text-2" id="text-org7f847ec">
</div>
<div class="outline-3" id="outline-container-org3b5ed85">
<h3 id="org3b5ed85">extract 移除整个tag</h3>
<div class="outline-text-3" id="text-org3b5ed85">
<p>
移除tag或String，返回被移除的。例如：
</p>
<div class="org-src-container">
<pre class="src src-python"><span style="font-weight: bold;">from</span> bs4 <span style="font-weight: bold;">import</span> BeautifulSoup
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">示例html：</span>
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">&lt;script&gt;a&lt;/script&gt;</span>
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">baba</span>
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">&lt;script&gt;b&lt;/script&gt;</span>
<span style="font-weight: bold; font-style: italic;">soup</span> = BeautifulSoup(<span style="font-style: italic;">'&lt;script&gt;a&lt;/script&gt;baba&lt;script&gt;b&lt;/script&gt;'</span>, <span style="font-style: italic;">'lxml'</span>)
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">查找所有的script标签，然后调用 extract 移除</span>
[s.extract() <span style="font-weight: bold;">for</span> s <span style="font-weight: bold;">in</span> soup(<span style="font-style: italic;">'script'</span>)] 
soup <span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">输出处理后的soup</span>
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">输出：</span>
<span style="font-weight: bold; font-style: italic;"># </span><span style="font-weight: bold; font-style: italic;">baba</span>
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="status" id="postamble">
<p class="date">Date: 2019-3-20</p>
<p class="author">Author: gdme1320</p>
<p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
